VOICE INTEGRATED SYSTEMS 
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NAVAL AIR DEVELOPMENT CENTER 
WARMINSTER, PENNSYLVANIA 


VRAS - A Voxce Recognition cind Synthesis System 

The program at NADC was initiated to determine the desirability 
of interactive voice systems for use in airborne weapon systems crew 
stations. To acconplish this effort, a voice recognition and synthesis 
system (VRAS) was developed and incorporated into a human centrifuge. 

The speech recognition aspect of VRAS was developed using a voice com- 
mand system (VCS) developed by Scope Electronics. The speech synthesis 
capability was supplied by a Votrax, VS-5, speech synthesis unit built 
by Vocal Interface. The effects of simulated flight on automatic speech 
recognition were determined by repeated trials in the VRAS-equipped 
centrifuge. The relationship of vibration, G, O 2 mask, mission duration, 
and cockpit tesperature and voice quality was determined. The results 
showed that: 1) voice quality degrades after 0.5 hours with cin O 2 mask; 

2) voice quality degrades under high (± 0.3G) vibration; and 3) voice 
quality degrades under high levels of G. The "voice quality" studies 
are summarized in Figure 1. These results were obtained with a baseline 
of 80 percent recognition accTiracy with VCS. 

The next phase of the develojanent program called for improve- 
ment of the VCS system. This was accomplished in two ways. A consis- 
tent bit was incorporated into the process wherein reference patterns 
are estcdslished to improve recognition accuracy. Improved recognition 
accuracy was noted. A syntactical haindler was developed to facilitate 
use of the isolated word VRAS system and to assist simultaneously in 
the :inderstanding process. The developed syntactical handler was tested 
with teletype input and was operational with 100 percent accuracy in 
real time. 

The major components of the VRAS system and its general oper- 
ation are shown in Figure 2. We see that the spoken words, originated 
by a speaker, are analyzed and sent to the "Statement Understanding" 
component. Once the meaning of the statement is understood, it is for- 
warded to the "Message Handling" unit which is responsible for all ex- 
changes of information between the VRAS system and the system computer 
to which it is interfaced. Having accomplished the intent of the speak- 
er's statement, the appropriate reply is created by the "Response 

^The opinions eipressed here are those of the author and do not necessarily 
reflect the official policy of the United States Navy. 
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Figure 1, Summary of Voice Quality Studies 
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VRAS SYSTEM 
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Figure 2. Voice Recognition and Synthesis (VRAS) System 




Generator" and then given to the original speaker via a "Voice Synthe- 
sis" unit. The "Visual Scanner" permits visual feedback to the speaker 
from a variety of other VRAS units which allows the speaker to monitor 
visually what is being said and understood by those various VRAS com- 
ponents. The VRAS system also includes a printer, card reader and 
disc drive for logging out all communications, inputting vocabulary 
data, and storing different speakers' trained words for word recognition 
purposes . 


An overview of the VRAS system, and the statement understand- 
ing approach it employs, is presented in a paper entitled "VRAS - A 
Voice Recognition and Synthesis System” which appears in volume VII of 
the 1976 SID International Symposium Proceedings. This paper was 
authored by Dr. Robert J. Wherry, Jr., who originated and developed 
the VRAS system. 

This system permits the use of medium-sized vocabularies 
(250 words) and highly flexible statement formats. Among the unique 
concepts featured in the VRAS system are: 1) a "universal" statement 

format, 2) the use of "truth" logic to eliminate words which can no 
longer be appropriate in the sentence being said; and 3) a "dictionary 
of meaning" which permits the VRAS user to communicate a given message 
in a large variety of different sentences. Since the syntactical 
handler only requires a recognized word or phrase as input, it can be 
used with recognition devices other than the Scope VCS . The flexibility 
of the VRAS system allov;s for the development of a syntactical handler 
to accomplish any specified application within a month. The value of 
the syntactical handler is that it will allow the user to vary the 
syntactical arrangement of words during data entry without affecting 
recognition accuracy. Thus, the natural quality of speech as a data 
entry means is preserved. 

The development of the VRAS facility and the VRAS concept 
has resulted in a powerful approach for accomplishing voice recognition 
and synthesis. However, since the the programming language used was 
at the assembly level, and since the computer employed was a Raytheon 
704, only the Naval Air Development Center could readily utilize the 
VRAS capability. Because of the use of assembly level language, changes 
and improvements to VRAS have proved extremely time consuming and costly. 
Because of the use of the Raytheon computer the VRAS approach has not 
been readily applicable to the requirements of other identified voice 
development efforts. To correct these deficiencies a work effort has 
been com.pleted which developed, tested and dociamented a FORTRAN IV 
packaging of the VRAS program. Program modifications or transferral to 
other computer systems or recognition and synthesis equipment have be- 
come simplified and readily implementable since all coding, except for 
the specific equipment interface routines, are in ANSI Fortran. Eight 
types of programs are required to ensure flexibility and inter-system 


126 


compatibility, as well as to accon 5 >lish the VRAS syntax processing 
f\inction. Seven system implementation programs are interfaced through 
a supervisory program that provides the few instructions required to op- 
erate the system. The programs, as shown in Table 1, can be run inde- 
pendently or through the supervisory program. A nrare detailed descrip- 
tion of the use of each program, and how it is accessed, is provided in 
a report which is available for general distribution. 

"Unlimited" Vocabulary Recognition and Understanding 

The thrust of the previously described VRAS program was to 
concentrate on understanding the meaning of what was being communicated 
rather than merely on the peurticulcir sequence of words which was employed. 
While the VRAS system does permit the use of medivim-sized vocabularies, 
and while it does permit a relatively flexible sentence stiructure, the 
greatest single drawback to the use of real-time voice recognition and 
synthesis today is still the limitation on vocabulary size and sentence 
structure. To understand the nature of this limitation the two major 
approaches to word recognition or "voice analysis" must be presented. 

One approach involves an analog-to-digital conversion of the input voice 
signal and a frequency analysis using bandpass filters to record what 
the voice signal was during a given period of time. Correlating the 
obtained and expected activity levels for the different bandpass filters 
over time permits word recognition to occxir. A second approach also 
uses time samples of activity for a bank of bandpass filters. In this 
case the patterns of activity in the filters are compared against a set 
of phonetic features to determine the presence or absence of various 
kinds of sounds (fricatives, stops, etc.). When using small, tailored 
vocabularies, both approaches tend to do a very good job of correctly 
identifying the actual word being said by the speaker. 

The concept of permitting the speaker to use an "unlimited" 
vocabulary - any legitimate English word - has been rejected as an unreal- 
izable near-timeframe objective for voice recognition and synthesis sys- 
tems because of the difficulty in word recognition for a relatively few 
words. It is not merely that as the niimber of words in the vocabulary 
increases the more probable it is that two words will sound alike; it 
is more than this problem, which we might call the "word confusion" 
problem, which has discouraged the development of truly "large vocabu- 
lary" voice recognition and synthesis systems. For exan 5 >le, for each 
word in the vocabulary, its "recognition vector" (the way the speaker 
has previously said that word) must be stored, and if that word is 
to be used as a synonym for cinother word, then its definition must 
also be stored. With the VRAS system, using the Scope device, 256 
bits of storage for each word in the vocabulairy were required just 
to store the recognition vector. Assuming 16 bit computer words, each 
recognition vector would require 16 computer words; a vocabulary 
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TABLE 1 


VRAS 

TRAINING 

PARSE 

VOCAB 

CONFUSIN 

RAWDATA 

DEMO 

RETRAIN 


VRAS SYSTEM PROGRAMS 


VRAS System Interface Program allows access to all other 
programs in the VRAS System. 


Trains VRAS to a specific speaker and vocabulary. 


Processes sentences entered by speaker. 


Lists the current vocabulary and subsidiary programs. 


Enables the user to determine an appropriate recognition 
correlation threshold and a list of possibly confusing 
words . 


Enables the user to obtain listings of both the short 
(processed) and long (unprocessed) recognition vectors. 


Demonstrates the VRAS training and recognition logic. 


Allows the user to add additional repetitions of the 
words in the vocabulciry to the composite recognition 
vectors stored on the disc. 
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of only 2000 words would require 32,000 words just to store the recog- 
nition vectors. Another accompanying problem with large vocabularies 
is the increased processing time required for the additional compari- 
sons to be made vrtien trying to determine which word has been said by 
comparing the "spoken vector" with the various "recognition vector." 
Thus, present word recognition technology cannot handle "unlimited" 
vocabularies for three very good reasons: 1) large vocabularies require 
too much storage, 2) leurge vocabularies require too much processing 
time, and 3) large vocabularies permit too many words which tend to get 
confused with each other because they produce too similar recognition 
vectors . 


While the aibove reasons would, at first consideration, seem 
sufficient to reject the concept of an unlimited vocabulary voice 
recognition and synthesis system, it will be seen that an alternative 
approach to word recognition may be possible. The alternative approach, 
which will be referred to as the "word-part" approach, is based on the 
concept that the vast majority of words used by speadcers are various 
combinations of relatively few prefixes, stems, and suffixes. If an 
incoming word can be analyzed into its component word-parts, not only 
can the word be correctly recognized, but also its appropriate meaning 
can be established without reference to a "dictionary of meaning." 

Just as the stem of the word has its own meaning, so also do 
the prefixes and suffixes. It, therefore, becomes possible to conceive 
of a new word recognition approach which analyzes each spoken word into 
its component prefixes, suffixes, and stems; to determine the meanings 
of these components; eind to use those component meanings to determine 
what the speaker is saying without ever attempting to recognize the 
whole word or to have a definition of the whole word stored in memory. 

This new approach to ein "unlimited" vocabulary voice recogni- 
tion and synthesis system will be pursued during fiscal years 1978 and 
1979 as an exploratory development effort which should complement and 
extend the previously described VKAS development program. 

Integrated Applications of Automated Speech Technology 

Progress in isolated word recognition, syntactical handling, 
and other speech technology areas provides evidence to suggest that 
the initiation of a Navy Advanced Development Program is justified. 

However, developments and progress in separate speech technology areas 
can only achieve their true potential if and when they are successfully 
integrated into total system applications. It is noteworthy that several 
such integrations have been achieved. The Naval Training Equipment 
Center's Ground Controlled Approach Controller Training System has uti- 
lized speech recognition to effect control of an aircraft/pilot simulation. 
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and to provide the basis for the objective performance measurement of 
the trainee ' s behavior . The Department of Transportation ' s Automated 
Command Response Verification System has demonstrated the integration 
of automated speech technologies (AST) in an operational ship-safety 
role. In these and other government applications AST has done more than 
make life a little simpler. A number of applications have successfully 
demonstrated the power of AST-based systems in solving problems that 
could not have been addressed before the emergence of these technologies. 
Therefore , it seems that an advanced development program which syner- 
gistically draws upon the results of past and present AST efforts is 
a reasonable and worthwhile next step. 

However, before such a program can be initiated and success- 
fully pursued, several information gaps must be resolved. Specific- 
ally, if AST is to be applied to the areas of airborne crew station 
design, performance measurement and training simulation, several new 
methodologies must be developed. They include: 1) a method for identi- 
fying high payoff applications of voice interactive systems in terms of 
the enhancement of both operator and system performance; 2) a methodology 
for assessing the technical feasibility of AST for each proposed appli- 
cation; and 3) a methodology for integrating the above information sources 
and generating a rationale for mutually supportive basic research, and 
exploratory and advanced development requirements. 

The Integrated Applications of Automated Speech Technology was 
an exploratory development program initiated in fiscal year 1977 to de- 
velop these methodologies. This program will be completed in early 
fiscal year 1978. The program objective is to develop methodologies 
for integrated applications of automated speech technologies for Navy 
system development, training, and operational settings. The program 
approach includes five major tasks: 1) review government applications 
of AST; 2 ) perform crew station design analyses; 3) examine performance 
measurement capabilities; 4) examine training applications; and 5) pre- 
pare a program/ implementation plan. 

The objective of the review task was to review critically the 
present applications of 'AST, and their supporting data, to establish a 
baseline of present progress from which the Navy can draw to plan AST 
applications. The completed review identified present capabilities and 
advancements, as reflected in successful system applications of AST, for 
type of speech recognition (i.e., isolated and limited continuous), 
vocabulary, recognition accuracy, syntactical handling, and user accept- 
ance . 


The crew station design, performance measurement, and train- 
ing applications tasks have addressed documentation available for the 
Navy P-3C anti-submarine aircraft weapon system to develop the desired 
methodologies. For the crew station design task the "on station" mission 


130 


segment was examined for each crew member by considering tasks to be 
performed and subsystems to be exercised- After consideration of the 
variables that affect the application of voice technology to crew sta- 
tion design, a four dimensional rating system was developed. The di- 
mensions included: the technical feasibility of implementing voice for 

accon^jlishing the task; the utility of implementing voice to accomplish 
the task; time/accuracy requirements for the task; and the impact of 
unassessed variables such as aircraft noise and mission duration. Using 
this rating strucutre, each task was reviewed utilizing the four dimen- 
sional requirements, euid assigned a four digit code of numbers corres- 
ponding to the four task requirement. For each task the four digit code 
was reduced to a one digit' code corresponding to initial AST payoff. 
Previously obtained criticality and frequency ratings for each task were 
applied to this initial AST payoff rating to obtain an overall AST pay- 
off rating. Finally the ratings for the tasks were converted to a matrix 
format. As an ..example of this process. Table 2 shows tasks by subsystems 
for the P-3C Pilot. The AST ratings for all detailed tasks to be in- 
cluded within a matrix cell were treated statistically to determine a 
single AST potential payoff rating for each matrix cell. Table 3 sum- 
marizes the most promising voice applications areas for both the P-3C 
Pilot and TACCO. It should be recalled that the objective of this task 
was not to identify voice applications for the P-3C, but to develop the 
methodologies required to identify high payoff applications of voice 
technology. 

As of this time the performance measxarement and training 
applications tasks are not completed. 

The last task of this effort involves preparation of a pro- 
gram/implementation plan. The general approach for integrating various 
information sources and generating a rational for research implementa- 
tion requirements is presented in Figure 3. Neither the various trade- 
off analyses nor the methodology for effecting the integration of the 
various information elements have been completed. Again the promise of 
this task is that the approaches developed for the generation of the 
trade-off analyses and the integration of the information sources will 
be incorporated when atten^ting to apply voice technology to other 
airborne systems/subsystems. The identification of technology voids and 
problems will serve as the basis of an interlocking technology base pro- 
gram covering the full spectrum of basic research through advanced 
development . 

VIST - Voice Interactive Systems Technology 

VIST is a new advanced development program being initiated in 
fiscal year 1978. It is viewed as the application or implementation of 
the products obtained from the previously described AST exploratory de- 
velopment effort. The objective of the program is to demonstrate the 
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VOICE TECHNOLOGY PAYOFF FOR P-3C PILOT 


TABLE 3 


P-3C PILOT/TACCO SUMMARY RESULTS 


PRELIMINARY ANALYSIS OF MOST PROMISING AREAS FOR VOICE APPLICATION 

A. TASKS (IN ORDER OF PROMISE) 

1. MONITOR INDICATORS (ALERTS) 

2. ACTIVATE SWITCHES (FUNCTION SWITCHES) 

3. ENTER DATA (KEYBOARD) 

4. ADJUST CONTROLS (KNOBS) 

5. RECEIVE DATA (CRT TABLEAUS) 

6. COORDINATE DATA (COMMUNICATION) 


B. 


SUBSYSTEMS (IN ORDER OF PROMISE) 

1 . COMMUNICATIONS 

2 . PROPULSION 

3 . SEARCH STORES 

4. PHOTO 

5 . DATA HANDLING 

6. ARMAMENT 

7. CREW 
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applicability of voice-based technologies to the specific areas of air- 
borne controls/displays design, perforacince measurement, and training 
simulation. The general purpose of the effort is to apply AST to obtain 
a carefully determined voice interactive system (VIS) which, when coupled 
with a multi-task simulator will; 1) reduce operator loading by sharing 
operator display/control functions between the visual/motor channels and 
the verbal/auditory channels, and 2) allow direct measurement of operator 
performance which will result in more precise control of the training and 
skill monitoring processes, and a more meaningful index of operational 
readiness. 


The general approach of this program involves: 1) incorpor- 

ation of the results of the exploratory development Integrated Applica- 
tions of Automated Speech Technologies effort to develop a strategy for 
demonstrating the suitability of applying voice to accomplish program 
objectives; 2) design, development, eind exercising of a voice system 
and simulator to allow for performance of selected airborne tasks, and 
the provision of a capability for recording and processing operator/ 
system voice transactions; 3) exercising the voice system and simulator 
to evaluate proposed task applications for cost effectiveness, contri- 
bution to system effectiveness, and operational acceptability; and 4) 
generation of detailed system specifications for the implementation of 
voice applications by program managers. 

A five year development effort is spelled out in the detailed 
program plan. Major milestones include: 1) determination of voice sys- 

tem and simulator functional requirements; 2) determination of a config- 
uration for the demonstration system; 3) preparation of the detailed 
work breakdown structure; 4) preparation of the detailed implementation 
plan; 5) development and integration of the demonstration system; 6) 
demonstration/evaluation of the system to determine the adequacy of each 
implemented task to fulfill functional requirements; 7) performance of 
required cost/benefit/effectiveness analyses; 8) generation of detailed 
design specifications for each major component of the voice system, e.g., 
a detailed design specification for a voice recognition element; 9) 
generation of a taxonomy of selected airborne tasks which can serve as 
a guide for the utilization and application of voice technology; and 10) 
transition to engineering development. This last effort calls for the 
identification of target or candidate systems, and system tasks, which 
promise high payoff for the application of voice technology. The ad- 
vanced development program will also include integration of the proto- 
type voice system with selected platforms to provide intermediate demon- 
strations of the utility of voice technology for target or candidate 
systems, and to provide for an orderly transition to engineering develop- 
ment . 


The generalized product of this effort is an intermediately- 
validated voice interactive system which has demonstrated its capability 
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for providing a solution for problems of airborne control/display design, 
performance measurement, and training simulation. Upon completion of 
the VIST program a data base consisting of; 1) a set of detailed de- 
sign specifications, and 2) a set of identified airborne tasks will be 
available to program managers responsible for the development of major 
airborne systems/subsystems. Such managers can match the tasks to be 
performed by the operators of their system or subsystem against those 
tasks which have been shown to contribute to enhanced system effective- 
ness by the application of voice technology. More important, program 
managers will have available a vehicle to accomplish the implementation 
of voice technology for their particular purpose. That is, they will 
have available detailed design specifications for voice interactive 
systems which, if identified as system requirements, will provide sound 
assurance that the developed system will best reflect the maximal con- 
tribution of voice technology. 

In addition, the voice integrated system and simulator capa- 
bility developed during the advanced development program can se 2 rve as an 
in-house capability to be exercised to support a program manager in de- 
termining whether a contractor-developed voice system adheres to system 
design and performance requirements. A program manager will have avail- 
able for his use a tool to accomplish independent system verification 
and validation. This approach has repeatedly demonstrated its value in 
the development of large-scale software and hardware systems. 

While the VIST program is directed toward implementation in 
airborne systems, the processes and products of this program should be 
readily utilizable for accomplishing the incorporation of voice tech- 
nology in complex operator stations in submarines, surface ships, and 
ground-based installations which require operator and system interfacing. 
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CDR Mike Curran, Ph.D. 
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Research Laboratory, Pensacola, FL. , where he contributed to the develop- 
ment of an on-line, interactive test system to be employed as a research 
tool and as an approch for implementing secondary selection procedures. 

He also pursued a long-term program concerned with the effects of psych- 
ological stress on performance. 
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DISCUSSION 


CDR Mike Curran, Ph.D. 


Q: Bob Hilgendorf, Reconnaissance Strike System Program Office at 

Wright Patterson : You mentioned that your FY78 VIST Program is 

a 6.3 effort. Was your FY77 program where you were developing some 
of these methodologies a 6.3 effort also? 

A: No, maybe it was confusing to you. The Integrated Applications of 

Speech Technology, i.e. the five tasks I described, will flow into 
our advanced development program. 

Q: Hilgendorf : Okay, so it’s a 6.2 effort? 

A: Yes. 

Q; Hilgendorf ; Who is doing the work? 

A: Who is doing the integrated applications work? Logicon and Boeing 

on a teamed contract. 

Q; Hilgendorf ; Okay, you were talking about the generation of some 
systems specifications as a fallout to your 6.3 program. 

A: No, as the product, not the fallout. 

Q: Hilgendorf ; Okay, as a product. Do you have any time factors as 

to when you have pitched to your people that you are going to be 
producing these specification? 

A: I would love to give that one to Cdr. Lane. Depending on the 

availability of funding, what is the current program schedule, 

Cdr. Lane? 

Cdr . Lane ; Fiscal Year 81 or 82. 

Q; Hilgendorf : Okay, then my last question is even though the deci- 

sions concerning the applications to certain airborne platforms 
are supposed to be made in an orderly fashion, do you have any 
teasers as to what systems you're talking about besides perhaps 
P-3C? 

A: My own private opinion? 

Q; Hilgendorf : Yes, that's good enough. 
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A; V/STOL-C. And only if you know that there is a V/STOL-A and 
a V/STOL-B-.- It's a pretty safe bet. We can make an intact on 
V/STOL-C, if there continues to be a Navy V/STOL Program. 

That time frame is approximately 1990. 

Q: Hilgendorf ; Okay, I don’t want to comment any further. 

Q: John Allen, MIT ; I wonder in the application you mentioned where 

you talked about computing the meainings of words from route 
words cuid prefixes and sxif fixes, what kind of vocabulary you have 
in mind auid do you have a basis for determining those meanings 
from the constituent morphemes of the word? 

A: Yes. I'll maJce one statement. Since this effort is in the process- 

ing of contracting now, I can't go into any more detail. We are 
not going to be dealing with the total unlimited English language 
dictionary of 80,000 words. Our effort will restrict itself to 
what we call the aviation dictionary. We think aviators do not 
utilize more than 10,000 words, whether they're Navy or Air Force. 

We have already gone though a College Dictionary and know there 
is a large number of words that are never used in aviation. We're 
talking, initially, of an unlimited vocabulary of 10,000 words. 

Those words, of course, include prefixes aOid suffixes. You may 
be talking about 1,000 to 2,000 parts, or what we'd call word 
roots . 

Q: Ken Woodruff, Systems Research Laboratories ; Cdr.' Curran, you 

talked about the Boeing effort. . , 

A: Boeing - Logicon effort. 

Q; Woodruff ; . . . . tried to develop a methodology for determing high 
payoff applications and I believe they are using the P-3C as 
their test bid for that. You did not get into the criteria 
by which they are making those payoff decisions. Would you 
care to comment on that? The reason I ask the question is the P-3C 
is a multi-crew vehicle. We have a severe problem within the Air 
Force community that we are going to all single seat aircraft. The 
priority structure might change considerably if that were your 
consideration. 

A; Even though Boeing aind Logicon are here, I think I can comment on 
that since our first decision in initiating this effort was what 
platform we were going to use. Obviously, in the Navy we have 
other than one-seat airplanes. But our decision regarding a csm- 
didate platform to develop these methodologies involved do we 
use fighter, do we use attack, do we use multi-engine? By the 
way. Commander Hanson is here. This effort is jointly sponsored 
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with the Office of Naval Research. There was considerable debate 
as to what type of platform we should examine. Should it be ASW, 
fighter, should we use helo or what? I think by brov±ieating or 
consensus , we decided on ASW because we in the Navy at NADC 
think we know something about ASW. Boeing has a wealth of 
experience in P-3 ' s and Logicon is close to the Navy environment 
and quite expert in voice systems applications. So we chose 
something we are familiar with, but also something that is well- 
documented. The P-C Charley is a well-decumented aircraft in terms 
of what the tasks are. The other approach was we could have chosen 
a future platform, and tried to anticipate tasks. It would have 
been more difficult. So, that's why we chose the P-C Charley look- 
ing at all operators and all tasks. 

Q: Unidentified Questioner (in distance) : Question not recorded. 

A; You are going to get into the complex meaning of the technology 

payoff, the utility. I brought copies of the rating process which 
describes in detail how v/e came up with the final "AST Payoff 
Rating." And I'll be happy to share it with you. Any other 
questions? 

Again, just let me emphasize just one thing. You saw final one- 
digit ratings and heard my verbal description which was cursory. 

You can realize that with the P-C Charley we had available a 
Coarseware documentation on criticality and frequency for operator 
tasks. That information, combined with Boeing and Logicon 's best 
judgment of what the technical feasibility of the application was, 
gave us the worth of the proposed. Any other questions? 
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